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Abstract. In this paper, we present a simple combinatorial proof of a Weyl type 
formula for hook Schur polynomials, which has been obtained by using a Kostant 
type cohomology formula for 0( m i„. In general, we can obtain in a combinato- 
rial way a Weyl type formula for various highest weight representations of a Lie 
superalgebra, which together with a general linear algebra forms a Howe dual 
pair. 



1. Introduction 

The notion of hook Schur polynomial was introduced by Berele and Regev [3], as 
characters of complex irreducible tensor representations of the general linear Lie su- 
peralgebra 0l m | ra . Recently, in jTT] Cheng and Zhang proved a Kostant type cohomol- 
ogy formula for g[ m | n associated to its irreducible tensor representations to compute 
the corresponding generalized Kazhdan-Lusztig polynomials (cf.jMj), which also im- 
plies a Weyl type formula for irreducible tensor representations by Euler-Poincare 
principle (cf.|18j). This Weyl type formula, which is given as an alternating sum of 
characters of Kac modules, is closely related with a general approach to the study 
of the complex irreducible finite dimensional representations of Ql m \ n (cf . [51 l22"j [M] ) . 

In this paper, we introduce a new combinatorial proof of the Weyl type formula for 
hook Schur polynomials obtained in [llj . Our proof, which was originally motivated 
by [12] , is simple and natural in the sense that we use only the classical Weyl formula 
and the Cauchy identity for Schur polynomials. In fact, we prove a Weyl type formula 
for a more general class of functions which arise naturally as characters of quasi- 
finite irreducible representations, not necessarily finite dimensional, of various Lie 
(super)algebras [U [I5j [III [23], and were also introduced in a combinatorial way in 
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Suppose that A and B are ^-graded sets at most countable, and A is a generalized 
partition of length d. Let 

11,1/ 

where S^x^) and ^(x^ 1 ) are super Schur functions (or super symmetric functions) 
in the variables x_4 = { x a \ a G A } and x^ 1 = { x^ 1 \ b G B } corresponding to skew 
shapes [i = (A + (p d )) /rj and v = (p d )/rj for some p > and r\ (see Definition 12. ip . 
Then the main result (Theorem I3.10p is 

Q A/B _ Z^wi-^Sx^MSx^-^B 1 ) 

where A A / B = ri|a.|=|6| C 1 _ x a x b 1 )H\a\^\b\( 1 + XaX^y 1 , W is a set of right coset 
representatives of an affine Weyl group of type with respect to a maximal para- 
bolic subgroup, and X w,± are defined under an action of w G W on A. We also give 
alternative proofs of a Cauchy identity of paired with rational Schur polyno- 

mials and a Jacobi-Trudi identity for (cf.[28]). Some of the arguments might 

be stated or understood more easily in the context of representation theory, but we 
give self-contained combinatorial proofs which do not depend on it. 

Now a Weyl type formula for hook Schur polynomials or irreducible tensor repre- 
sentations of fll m | n (Theorem 14. 3p follows as a byproduct, up to a multiplication of 
a monomial, when A and B are finite sets (say \A\ = n and \B\ = m) of even and 
odd degree, respectively. This recovers in a purely combinatorial way the character 
formula given in [llj. We also give another proof of the factorization property of 
hook Schur polynomials (cf. [H [33] ) . 

In general, we can obtain Weyl type character formulas for other irreducible high- 
est weight representations of a Lie (super)algebra, whenever it forms a Howe dual 
pair with a general linear algebra (cf . [8j [9j [T71 [19l [23] ) , since the characters of the 
associated representations are given by under suitable choices of A and B [28] . 
We discuss in detail one more example in representation theory when both A and B 
are finite sets of even degree (say |«4| = n and \B\ = m). We deduce from Howe du- 
ality [191 [25] that the corresponding , up to a multiplication of a monomial, is 
a character of an infinite dimensional representation of Qi m+n , which is of particular 
importance in the study of unitary highest weight representations of the Lie group 
U(m, n) (cf.|15j). In this case, we obtain a Weyl type formula given as an alternating 
sum of characters of generalized Verma modules (Theorem I4.10p , which recovers the 
Enright's character formula [13] with a different parametrization of highest weights 
for generalized Verma modules, and also an analogue of the Jacobi-Trudi formula 
for these infinite dimensional representations. 
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Finally, we would like to mention that the similarity of character formulas for 
g[ m | n in [IT] and Ql m+n in [14] was already observed, and a more direct connection 
between the Grothendieck groups of module categories of Ql m \ n and Ql m+n has been 
established in [TO] recently. 

The paper is organized as follows. In Section 2, we recall some basic terminologies. 
In Section 3, we derive a Weyl type formula, a Cauchy type identity, and a Jacobi- 
Trudi formula for S^ B . We also discuss a factorization property for when A 

is a finite set of even degree, which provides another proof of the factorization of 
hook Schur polynomials. In Section 4, we discuss applications to irreducible tensor 
representations of 0t m | n , and infinite dimensional representations of Ql m+n - 

Acknowledgement Part of this work was done during the author's visit at Na- 
tional Taiwan University in 2006 summer. He would like to thank Shun-Jen Cheng 
for the invitation and kind explanations on his recent works. 



2. Symmetric function 

Let us recall some terminologies (cf . [30J ) . A partition is a non- increasing sequence 
of non-negative integers A = (\k)k>i such that J2k>i < °°- The number of 
non-zero parts in A is called the length of A denoted by ^(A). We also write A = 
(l mi , 2" 12 ,...), where rrii is the number of parts equal to i. We denote by 3? the set 
of all partitions. A partition A = (Xk)k>i is identified with a Young diagram which 
is a collection of nodes (or boxes) in left-justified rows with A^ nodes in the kth row. 
We denote by A' the conjugate of A. For A,/i <G let A + fi = (A& + Hk)k>i, and 
if A D fi (that is, A^ > fi^ for all k), let A//J, be the skew Young diagram obtained 
from A by removing fi. 

For a set K which is at most countable, let Ak be the ring of symmetric functions 
in the variables xj^ = { \ k £ K }, and s\(x.k) the Schur function corresponding 
to A € & . When K is infinite, let uk be the involution on Ak, which sends s\(xk) 
to sa'(x^). 

Throughout the paper, we denote by A = Aq U A\ a ^-graded set, which 
is at most countable. For a E A, \a\ denotes the degree of a. We put Z = 
{0,±1,±2,...}, Z >0 = {1,2,...}, Z <0 = {-1,-2,...}, [n] = {l,...,n}, and 
[— n] = { —1, . . . , — n } (n > 1), where all the elements are assumed to be of degree 
(or even). Also, we define A' = { a' \ a G A } to be the set with the opposite 
Z2-grading, that is, \a'\ = \a\ + 1 (mod 2) for a € A. 
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Let x^ 1 = { x^ 1 | a G A } be the set of variables indexed by A. For a skew Young 
diagram X/fJ,, a super Schur function corresponding to A//i is denned to be 

(2-1) S x/fl (x A ) = s ^( x A)) s a'/^'( x a) 

fiCvCX 

(cf . [3j [30] ) . For simplicity, let us often write S^, = Sx/^i^-A)- When A is finite, 
is a hook Schur polynomial introduced by Berele and Regev [3]. Following our 
notation, we may write 5^ = X^e^» ^/^^AAt ^ or ^ e ^> anc ^ nence ^a* = s a(xu) if 
A = Ao, and = sy(x^) if ^4 = Ay. 

For a positive integer d, let Z^. = { A = (\y, ■ ■ ■ , A^) G Z d [ Ai > • • • > A^ } be the 
set of generalized partitions of length d. Put 

A+ = (max(Ai,0), . . . ,max(A d ,0)) G 
(2.2) A" = (max(-A rf ,0), . . . , max(-Ai, 0)) G 

A* = (-X d ,...,-Xy)EZ d + . 
The addition on Zl is defined in a usual way, and then A = A + + (A - )*. 

Definition 2.1 (|28j). Let A and B be Z2-graded sets, which are at most countable. 
For A € Zi., we define 

Sa(x.a;x£s) = ^5 , M (x^)5 ! ,(x B 1 ), 

where ^ and v are skew Young diagrams of the form 

f,= (x+(p d ))/ri, u={p d )/ V 

for some non- negative integer p and partition rj such that A + (p d ) G & and r/ C 
A + {p d ), (p d )- Let us write S^ B = Sa(x^;xb) for simplicity. 

Remark 2.2. (1) If B is empty, then S^ 23 is non-zero only if A is an ordinary 
partition, and in this case, we have = S^. 

(2) By definition, sf B can be regarded as the character of certain bitableaux. 
A combinatorics of these bitableaux, including analogues of the Schensted inser- 
tion, the Littlewood-Richardson rule, and the Robinson-Schensted-Knuth correspon- 
dence, are given in [28] . 

For A G let sa(xui) be the rational Schur polynomial corresponding to A, 
that is, SA( x [d]) = (^i ' • ' x <i) _P ' s A+(p d )( x [rf]) for p > such that A + (p d ) G 3?. For 
/u, v G we have 

( 2 -3) s„(x M )«„(x M ) = ^2 c ^ s A( x [d])> 
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where c^,, is a Littlewood-Richardson coefficient. Note that c^„ = c X+ [^ p r it q \ } rfN for 

all p, g > 0. Then we have another expression of Sf 6 linear combination of 
the products S^x^S^x^ 1 ) for v G 

Proposition 2.3 ([28]). For A G Z^_, we We 

S'a(x^;x b )= ^ c^,5 At (x^)5 i ,(x B 1 ). 

i(jj),e(y)<d 

□ 



3. Weyl type formula 

3.1. Main result. Put Z x = Z\{0}. Let P = © ieZ x Ze^ be the free abelian group 
generated by {e^ | i G Z x }. For i 6 Z, let be the transposition on Z x (hence on 
{ e, | i € Z x }) given by 



(3.1) r, 



+ if « > 0, 
(i — 1 i), if i < 0, 
[(-11), if i = 0. 



Let W be the Coxeter group generated by { r.- L j i G Z }, and £(u>) denotes the length 
of w £ W. For each subset J C Z, let Wj be the subgroup of VF generated by 
{ rj | i 6 J }. Let 

(3.2) W = {weW\ l{nw) > £{w) for i G Z x }, 

be the set of right coset representatives with respect to a maximal parabolic subgroup 
W Z x (cf.0]). 

For A G Zi and w G W, choose sufficiently large p, q > such that 

(1) -P < Xd < Ai < g, 

(2) it) G W^(p, g ), where I(p, q) = {k G Z | -p+l<A;<g-l}. 

Put jtt = (A + (p d ))' = (pi, . . . , /i ra ), where n = p + q. We may identify /i with 

A* = Vie-p H h /U p e_i + //p+iei H h Pp+qe^ G P. 

Then, we define 

(3.3) woA = u)(/i + <5 P) g) - - dl~, 

where <S M = Ei 6 [-p](9 ~ * ~ 1 ) e ' + Eie[ g ](? ~ i) e i> and V = Eie[- P ] e i- 
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Lemma 3.1. Under the above hypothesis, there exist unique a, r G & with £(a) < p 
and £(t) < q such that 

wo\ = - ^ o--i6i+ Y T J e i- 

Proof. First, let w(5 P! g) — 5 P ^ = Yliez* a i e i e P-> where a, = for i g" [— p] U [q]. It 
is not difficult to see that w(5 p ' tq i) — 5 P > <q i = w(5 P:q ) — 5 P:q for all p' > p and q' > q. 
Since w is a right-coset representative of W%* in W, we also have ai > ai + \ and 
a_,;_i > a_j for all i > (cf.[l]). This implies that 

> a^ v > a_„+i > . . . > a_i, 

(3.4) 

«1 > Gt2 > • • • > a 9 > 0. 

Next, if we put w([i) = Ylie[-p]u[g] ^* e *' then we have 

(3.5) d > b- p > ■ ■ ■ > b-i > 0, bi > ■ ■ ■ > b q > 0. 

Note that w(fi) — dl~ does not depend on the choice of p, q. Hence it follows that 

w ° A = Ei6[-p]U[«] C * 6 *' Where 

> C_r, > Cl_„ > . . . > C_l, 

(3.6) 

Cl > C2 > . . . > C q > 0. 

This completes the proof. □ 
Definition 3.2. For AeZj and w G W, we define 
(3.7) X w '- = a', X w ' + = t', 

where a, r G ^ are given in Lemma 13.11 

Remark 3.3. (1) Given w G W, suppose that u; G for some p, g > 0. If 

w{5 p ^ q ) — 5 PtQ = X^ez x ^i e i e then we can check that the partition {—^-k)k>i is 
the conjugate of (fJ-k)k>i (see 2.4 in [4]). Moreover, the map sending ui to (nk)k>i is 
a one-to-one correspondence between W, the set of the minimal length right coset 
representatives and where £{w) = = Ylk>i^k- 

(2) One can also check that for A G Z d + , (\ w >-, \ w >+) = {X w '^, X w '>+) if and only 
if w = w' G W. 

Next, consider the Schur polynomials in n variables. Fix p, q > such that 
p + q = n. Instead of [n], let us use [— p, q] = [—p] U [q] as an index set for the 
variables. For a partition [i with £(fj>) < n, we may identify (i with 



me^ p H h /U p e_i + ^p+iei H h /ip+g£ g G P. 



Then Wj(j,^, which is isomorphic to the symmetric group on n letters, naturally 

acts on \i. Given a = YUe[-p,q] c * e * G P > P ut X [- P .q] = ^ie[-p,q] x ? • Reca11 triat the 
Weyl formula for the Schur polynomial corresponding to fi is given by 

(3-8) s^ p , q] ) = f^gW^j X [-^ . 

Then we have a parabolic analogue as follows. 

Lemma 3.4. Suppose that fi C (d n ) for some d > 0. Following the above notations, 
we have 



-,/i„ , , _ E we w nvy I(p , g /- 1 ) /Wa (A«.+y(x[ g |)g(A«.-y(X[_p]) 
x [- P , 9 ] s M x [-p,g]J - 



rLe[-p]nje[g]( 1-x i x i) 
where A = // — (p d ) S Z+ and A™' are defined in (j3.7j) . 

Proof. By Lemma [3TTI we have for ui G W n W/(p )9 ), 

ie[-p] je[<?] 

for some cr, r £ ^ with ^(tr) < p and £(r) < g. Now, for w' € Wj(p,g)\{o}j we have 

w'w(p, + 5p, g ) - 5 Pi q - dl p 
= w' (w(fi + 6 Piq ) - 5 p , q - dip + 5 Ptq ) - 5 p , q 

(3.9) = w/ - a - i€i + Yl T 3 € i +Q 1 p+ S p+ s }\ ~ l 1 ^ ~ S p ~ 5 t 

\ ie[-p] je[q] J 



w' 



Y a-i6i + T i e i + S p + K ) ~ 6 p ~ S Z> 

*e[-p] je[<?] 



where 5 p = Eie[-p]( _i ~ 1 ) e i' = Ej6[ g ](?-i) e j and V? = 9^p + 5 P + S g- Since 
s^(x[_p]) = 8 v *(x.7_,) for G Z^., we obtain the result from (|3.8p . □ 

In terms of S^ S , Lemma 13.41 can be written as follows. 

Lemma 3.5. For X G WA, choose p,q > such that —p < A^ < Ai < q. Then we 
have 

M'/I-pY _ ^^nw I(Piq) (- i y iw)s x w '+( x lq}') s x^'-(^[- p y) 



S 



I\ie[- P y llje[q]'(l _ x i lx j) 
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Proof. Put fi = (A + (p d ))'- Note that S v (x[ q y) = s v '(x[ q y) and S v (x, p y) 
s r? /(xr 1 ,,) for T) € Then, we have 



X [-^]' S M( X [- P ,g]') = Z) x [-^]' s ^ x [-p]') s m/"( x M') 



= XT S f-(dP)( X [-p]') S l M/iy( X [q]') 
v 

( 3 - 10 ) = E s (*)/"( x h P ]' ) ^( x M') 

V 

= Yl s (p d )/^ x i- P ]^ s {x+(p d ))/v^iiy 

V 

The result follows from Lemma 13.41 by replacing X[_ p (J ] with X[_ p (? ]/ 
Proposition 3.6. For A € Zl, we Ziaue 



Z' >0 /Z' <0 _ E w gw(- 1 ) £( "' ) ' g A^.+ (xz' >0 )'gA^.-(x z / o ) 



□ 



and a Cauchy type identity 

n(i+xrV)( 1 +^)= E ^' >o/z ' <0 «A(z [d] ), 

where i £ ^<0' ■? e ^>0> ^ e an( ^ z [d] = { z fe I ^ ^ M }• 

Proof. First, it is easy to see that for p, q > ^ p ^ reduces to ^ p ' 

when we put = and aCi_i = 0. Hence, ^ p ^ has the well-defined limit 

i z ; /z' 

with respect to both X[ ? ] and xr , when we let p, q — > oo, which is equal to >0 <0 . 

The first identity follows from Lemma 13.51 

Next, consider the following dual Cauchy identity (cf.[30j). 



(3.11) 



n n( i+xi ^) = yi s A*( x w)v( z [«fl)- 

ie[n]je[d] fj,C(d n ) 



Choose p, q > such that p + q = n. Replacing [n] with [— p, q]' and multiplying 
{x-pi ■ ■ ■ x^y)~ d (zi ■ ■ ■ Zd)~ p on both sides, we have 

n n Hii+vrvw+wk) 

ie[-p]' je[q]' ke[d] 

(3 12) = X F-P] f s (a + W)'( x [-m]')sa(z[^) 

= sf /[ - p] 's x (z [d] ). (see ») 

Hence, by letting p,q — > co, we obtain the second identity. □ 
Proposition 3.7. For A G Z+, we have 

S Z >0 /Z <0 _ E m gw(~ 1 ) f( '" )g A'°-+(xz>o)>S'A'»>-( x ^ ) 



and a Cauchy type identity 



_ _ c z >o/ z <o _ r„ \ 



where i G Z<o, j G Z>o, ^ G [d]. 

Proof. Applying both ^z' <0 an d in Proposition 13.61 and then replacing x z / >q 
(resp. *-z> <0 ) by x^ >0 (resp. x^ <0 ), we obtain the identities. □ 
As a special case of Proposition 13.71 we obtain the following Cauchy type identity 
with a restriction on the length of partitions. 

Corollary 3.8. For d > 1, we have 

£ *a(x>a(v) = = ' d , 

^(A)<d 

w/iere x = x z>0 , y = y z>0 , and d = (0, . . . ,0) G Z^. 

Proof. Consider < 5 , ^>°/ z<0 - Replacing x^ q by yz >0 (that is, = for A; > 1), 
we have 

(3-13) S%> o/Z <° = £ S ,(x) Sl/ (y) = £ SA (x) SA (y) 

v={k d )/r} t{X)<d 
k>0, riC(k d ) 

since s/ fe dw r? (x) = s A (x), where A = (£; rf ) + 77* G ^ (we regard 77 as an element in 
Zl) by the Littlewood- Richardson rule. Combining with Proposition 13.71 we obtain 
the identity. □ 
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Remark 3.9. Corollary 13.81 can be stated more explicitly. Given w G W, let [i = 
(a\(3) be the corresponding partition (see Remark I3.3f) given in Frobenius notation 
with <5(/i) the length of a or (3. Then it is not difficult to see that 

0^+ = M +(/W) ; 0^- = M / +(d 5C " ) ). 
Hence, we obtain an alternative expression 

E Aie ,^(- 1 ) lAt|s / ,+(^(M))(x)v+(^(M))(y) 



(3-14) £ 



£(A)<d 



Theorem 3.10. For A G Z+, we /ictt>e 



C ^/B _ E wg w(-l) <?W ^. + (x^)^.-(x B 1 ) 



M/B 

where 

^ _ 0|a| = |6| C 1 ~ x a x b ) 

Iljal^lfc^ 1 + *) 
VFe also have the following Cauchy type identity 

nUaeA 1 ( 1 + x a^k)UbeB 1 ( l + x b lz k 1 ) <yVB , v 

fc€ [d] llae^iol 1 ^fcJll&eBcA 1 x b z k ) x ^ 

Proof. For convenience, let us assume that A C Z>o and B C Z<o with arbitrary 
Z2-gradings. Let A° (resp. B°) be the set of all positive (resp. negative) integers 
with a Z2-grading such that A° (resp. B°) is infinite and Ai C A° (resp. £?j C B°) 
for j G 1,2- 

We may view A^ >0 (resp. A^ <0 ) as a subring of A^ ( ° (g> A_4° (resp. Ajg° Cg> Ago). 
Applying u;.,^ and wgj to 5^ >0 / z<0 anc [ the Cauchy type identity in Proposition 13.71 
(we assume that the set of variables in A^ <0 is ), we obtain 

Q A°/B° _ I]u.gw(~ 1 ) £( "' ) ' S 'A m .+ ( x .A°)'S'A".-( x g° 1 ) 

^A°/B° 

nYiaeAi^ + x a z k)I\beB°(^ + x b lz k 1 ) sr^ r,A°/B° , \ 
„ (1 _ x -i z -i ) = 2^ S x s a(z n ). 

Finally, by letting x a = x^ 1 = for a G .4° \ A and 6 G 0° \ B in (|3.15j) . we obtain 
the results. □ 

Remark 3.11. The Cauchy type identity in Theorem 13.101 was also proved in a 
bijective way in terms of tableaux [28], which can be viewed as an analogue of the 
Robinson-Schensted-Knuth correspondence |27j . 
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3.2. Factorization property. Let us consider a particular case, where we have a 
factorization property of S^ B . Recall the following Cauchy type identity for skew 
Schur functions. 

Lemma 3.12 (cf.[30j). For A, p G & } we have 

s pm( x ) Wy) = ff — n \ 2 Vr( x ) s vr(y), 

pG^ llij>l^ re* 

□ 

By similar arguments as in Theorem I3.10I it is straightforward to rewrite the 
above identities as follows. 



Corollary 3.13. For A,/i £ we have 

ri|tt|^|6| 

~9> W\a\ = \b\( l - X aXb) 



ST> Q A oB _ ^}M£b\0- + x a x b) s-^ . B 
Z> ^P/A^p/M - n, , ,.,M - ; n-A ^ V^A/t- 



□ 



Lemma 3.14. Given A G and m G N, assume that d > m and X rn > 0. Let 

p > and p G ^ 6e suc/i i/iai A + G & and p C A + (p d ), TTien 

s (A+( p d))/p( x M) = s a+( x h) s ^/m( x m)' 
w/iere ^ = (p d ) + (A - )*. 

Proof. Note that Sp/ T (xr m i) corresponding to a skew Young diagram p/r is the 
weight generating function of SSTr m -i(p/r), the set of all semistandard tableaux of 
shape p/r with entries in [m] (cf.[T6| I3T)]). 

Suppose that SST[ m ] ((A+ / p) is not empty. As usual, we enumerate the 
rows (resp. columns) in A + (p d ) from top to bottom (resp. left to right). Also, we 
may assume that vj p is not empty, where v = (p d ) + (A - )*. 

For T G 55T[ m ] ((A + (p d )) /p), let T\ (resp. T 2 ) be the subtableau obtained from 
the columns of T with indices greater than p (resp. less than or equal to p). The 
shapes of T\ and T 2 are A + and v/p, respectively. So, this defines a map 

(3.16) SST [m] ((A + {p d )) /p) — > 5ST M (A + ) x SST [m] (u/p) , 

by sending T to (Ti,T2). 

We claim that this is a one-to-one correspondence, which establishes the corre- 
sponding identity of Schur polynomials. Let us construct an inverse of the above 
map. Given a pair (Ti , T 2 ) G SST^ ( A + ) x SST^ (y/p), we obtain a tableau T (not 
necessarily semistandard) of shape (A + (p d )) / p, where the first p columns form T 2 , 
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and the other columns form T\. Note that the row and column numbers are those 
in A + (p d ). 

For 1 < i < d, let aj (resp. 6j) be the entry placed in the ith row and the pth 
(resp. {p + l)th) column of T. We assume that either aj or hi is empty if there is 
no entry. Then b\ , . . . , bd are the entries in the first column of T\ , and a\ , . . . , ad are 
the entries in the last column of T2. 

Since m < d, we have b m+ i = ■ ■ ■ = bd = 0, and bk > k for 1 < k < m. If all 
dj are empty, then it is clear that T is semistandard. We assume that there exist 
non-empty entries a s ,. . . ,at with 1 < s < t < d. Since A m > 0, it follows that the 
pth column of A + (p d ) has at least m boxes, and hence (A + (p d )) / ' \x has at least 
one box in rows lower than or equal to the mth row. This implies that t > m, and 
Ofc < m + k — t < k < bk for s < k < t. Therefore, T is a semistandard tableau 
of shape (A + (p d )) //U, and the map (T\,T2) 1— > T is the inverse of A3. 16H . This 
completes the proof. □ 



Theorem 3.15. Suppose that A = [m] for m > 0. For A £ we have 

lyi 



S [rn]/B = S x+ ( X[m] )S X -{yi B 1 )A * 



if and only if d > m and A m > 0. 

Proof. First, note that by the Littlewood-Richardson rule, we have Sfi = S^l for 
any A and /j, E where \jF is the Young diagram obtained from \i by 180°-rotation 
(cf.[30]). Now, suppose that d > m and A m > 0. Then we have 

alm]/B 

x 

s At (x [m] )S' ! ,(x B 1 ) 

^={\+{p d ))h, 
v=(p d )h 

s A+( x M) s ((p [i )+(A-)*)/^( x M) 5 '^( x B 1 )' (by LemmaEUD 

n=(x+(p d ))/v, 
(3.17) v={p d )h 

= SA+( x [m]) V/A-^mlj^fxg 1 ) (by 180°-rotation) 

u=(p d )/r] 

= SA+( X [m]) Y S r/A-(x[ m ])5' r (x B 1 ) 
A-Cr 

= S 'A+( x [m])' S, A-( x B 1 ) A [^]/B Corollary EISD . 

Conversely, suppose that either rf<mor A m < 0. Let p > and r? € ^ be such 
that A + (p rf ) G ^ and 77 C A + (p d ), (p d ). Then it is not difficult to see that the 
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difference 



(3.18) s A+ (x H )s( (pd)+(A _ ) ») /j) (x H ) - S( A+(pd) ) /j? (x [m] ). 



is non-zero, in fact, an integral linear combination of monomials in xr TO i with non- 

[ml IB 

negative coefficients. It implies that the difference of S x and the last term in 
(|3.17p is non-zero. This completes the proof. □ 



Remark 3.16. We can also prove Theorem 13.151 using Theorem 13. 101 Suppose that 
A G Z^J_ is given. If d > m and A m > 0, then it is not difficult to see that £(X W,+ ) < m 
if and only if w = 1 G W, which implies that S ™ = S\+ (x[ m y)S\- (xg 1 )A^^. 
Conversely, if d < m or A m < 0, then we can check that there exists at least one non- 
trivial element w G W such that £(X W,+ ) < m, which implies that 5 A m ^ S Ar m ]/ s — 
S' A +(x[ m ])5 A -(xg 1 ) is non-zero by Remark 13.31 (2). 

3.3. Jacobi-Trudi formula. Finally, let us present another proof of the Jacobi- 
Trudi formula for [28] using the arguments given in 3.1. 

Given A and B, consider 

(3.i9) h«*m = n^^+^n^a+^l) = E ^ 

UaeAoi 1 ~ ^UbeBoi 1 ~ X b 1 ) t& 

where hf B = sf B = J2 m -n=k 5' m (x^)5 n (xg 1 ) for k G Z. 
Proposition 3.17 (cf.[28])- For A G Z^, we Ziaue 

a - ^(^-i+in^j^- 

Proof. Choose p, g > 0. Let efc(x[_ Pj(? ]/) be the /cth elementary symmetric polyno- 
mial in variables X[_ p q y for A; > 0, and efc(x[_ p >(? ]/) the coefficient of t fc for — p < k < q 
in 

n n( i +^ i )(i+^)- 

ie[-p]' je[q]' 

Then, from the classical Jacobi-Trudi formula, we have 

( 3 - 20 ) = (x-p> ■ ■ ■ x_r)~ d det(ex l+ p-i+j{-X[-p^y))i<i,j<d 

= det(e A< _j + j(x[_ P)? ]/))i< i j< c i. 

If we follow the same arguments as in Theorem 13. 10[ then ^ p ' and efc(x[_ p g ]/) 
are replaced by and h^ B , respectively. This completes the proof. □ 
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4. Applications 

In this section, we discuss applications of our Weyl type formula for in 
representation theory. We assume that the ground field is C. 

4.1. Representations of gl m i n . First, let us derive a Weyl type formula for hook 
Schur polynomials. Though its proof is purely combinatorial, we will translate 
the result in the language of representation theory to see its relation with finite 
dimensional irreducible representations of 0t m i n . So, let us give brief review on 
representations of the Lie superalgebra gl m \ n (cf.[20j). 

For non-negative integers m and n, not both zero, let C m ' n = C m '° C°' n be 
the (m + n)-dimensional superspace with the even subspace C m '° = C m and the 
odd subspace C°l n = C n . We denote by {e* \i G [— m] } and { Cj [ j G [n]' } the 
homogeneous bases of C m '° and C°' n respectively, which form a standard basis of 
C m|n Then the space of C-linear endomorphisms of C m ' n is naturally equipped with 
a Z2-grading, and becomes a Lie superalgebra with respect to a super bracket, which 
is called a general linear superalgebra gl m \ n - Put [—m,n'] = [— m] U [n]'. 

We may identify q = gl m \ n with the set of (m + n) x (m + n) matrices with respect 
to the standard basis of C m ' n . Then the subspace f) of diagonal matrices forms a 
Cartan subalgebra, and under the adjoint action of () on 0, we have a root space 
decomposition, = f) © (© Q( =A0a)> where A is the set of all roots of g. Let A + be 
the set of positive roots, and Aq" (resp. Af) the set of positive even (resp. odd) 
roots. Since I)* = C m ' n , we have 

(4.1) 

A+ = {e i -e j \ie [-m], j G [n]'}, 

where we assume that — m < —m + 1 < • • • < — 1 < V < 2' ■ ■ ■ < n'. We consider a 
Z- grading = 0-i©0o©0i consistent with its parity, where 0o = gl m © gl n , and a 
triangular decomposition, = n _ © () © n + such that g±\ C n 1 * 1 . 

Let P m \ n be the Zdattice of f)* generated by { €j | i G [— m, n'] }, which is called the 
set of integral weights, and let Pi n be the set of weights A = Y^ie[- m n'] ^ ^»n|n 
such that A_ m > . . . > A_i and Ai> > . . . > A n /. An element in -P^| n is called a 
dominant integral weight. 

Suppose that M is a finite-dimensional 0[ m | n -module, which is (j-diagonalizable. 
Then, we have a weight decomposition M = ©^ e f,* M\. When dim Ma 7^ 0, we call 
A a weight of M. For convenience, we assume that all the weight are integral. We 
define the character of M by chM = J^x^p dimA/^e A , where { e A I A G P m \ n j is 
the set of formal variables. 

Given A G P^ n , let L (A) be the finite dimensional irreducible highest weight 
0o-module with highest weight A. We may view L°(A) as a 0o © 0i-module, where 
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01 acts trivially on L°(A). The Kac module K m \ n (A) is defined to be the induced 
representation K m \ n (A) = U(g) ®u( Si) (B 9 i) L °W where U(q) and U(g gi) are 
the enveloping algebras, and it has a unique maximal irreducible quotient L m u(A). 
Then {L m | n (A) [ A E P^\ n } forms a complete set of pairwise non-isomorphic finite 
dimensional irreducible representations of Ql m \ n with integral weights. 

In [211 [22], Kac gave a necessary and sufficient condition for K m \ n (A) (A E P^i n ) 
to be irreducible, and called such weights typical. For an atypical weight A E Pi n , 
which is not typical, L m | n (A) has a resolution, where each term has a filtration with 
quotients isomorphic to Kac modules (see [33]), and its character is given by 

(4.2) chL Hn (A)= 

A'£P+, 

m | n 

for some oaa' G In [33], Serganova gave an algorithm for computing these coeffi- 
cients using the geometry of the associated supergroups. Recently, in [5j, Brundan 
gave another algorithm using a remarkable connection with canonical bases of quan- 
tum group Uqiglae). 

Now, let us give an explicit expression of (|4.2j) for irreducible tensor representa- 
tions. For A = J2ie[-mri] ^* e « e P^ n , we may identify A with a pair of generalized 
partitions given by 

(4.3) A <0 = (A_ m ,...,A_ 1 ) EZ™, A >0 = (A ls ...,A n( )GZ n + . 

If we put Xi = e 6i for i E [— m], and yj = e e j' for j E [n], then the character of 
K m\n( A ) is given by 

(4.4) chK m \ n (A) = s A <o(x[_ m ])s A >o 

ie[— m], 
j'G[n] 

Let be the set of dominant integral weights A such that A& > for all k E 

[—m,n'] and £((A >0 )') < A_i. For A E Pi n > we define a partition 

(4.5) A(A) = (A_ m ,...,A_ 1) Ai,...,Ai), 

where (A >0 )' = (A' l5 . .. , A' e ) and £ = e((A >0 )'). Then the map A i-> A (A) gives a 
one-to-one correspondence between V^ n and <^ m | n = { A E | A m+ i < n }, the set 
of all (m, n)-hook partitions. 

In [3J, it was shown that the tensor algebra ^(C m ' ra ) generated by the natural 
representation C m l n is completely reducible, and for A E P*i n , L m \ n (A) occurs in 
^(C m l n ) if and only if A E Moreover, we have 

(4.6) chL m , n (A) = s IJ,( x [-m]) s X'/ l j,' (y[n]) 
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with A = A(A) G ^mjni which is called the hook Schur polynomial corresponding to 
A. Following our notation, chL m | n (A) = s\ m ' n ^ = S>(xr_ m)Tl /]), where we identify 
x [n]' C X[_ TO) „/] with y[ n j. 

Lemma 4.1. For A G Pi n , Zet A = A(A) G ^ m | n a™d v = X' - (m d ) for d > X\. 
Then v € Z+, and 

(x_ m ---x_i) _d chL m | n (A) = S v (x[ n y,X[_ m y), 
where we identify xr_ m i with xr_ m i/, and y[ n ] with xr n i. 

Proof. By similar arguments as in Lemma 13.51 (|3.10p . we have 
(x_ m • • • x_i) _d chL m | n (A) 

= (x^ m ---x_ 1 y d 5^s M (x[_ m ])s A // M /(y [n ]) 

= ^ S^-(d™)( x [-m])sA'/^'(y[n]) 
^CA 

/iCA 

= Yl S (rn d )/n(*l-m}>) S (v+(™ d ))/v( y l n O by re P lacin g X [-m] with x [-m]' 
= ^(x[ n ];x[_ m ]/) = SrW/r-H by replacing y [n] with x [n] . 

□ 

We have another proof of the factorization property of hook Schur polynomials, 
or the irreducibility of the Kac-module K m \ n (A) for A G T^IrT 

Corollary 4.2 ([21 [33])- For A E T 7 ^; we /iai>e L m | n (A) = K m | n (A) i/ and only if 
A_i > n. 

Proof. For A G Pi n , A_i > n if and only if A = A(A) G =^ m | n contains the partition 

(n m ) of a rectangular shape. By Lemma liTTl chL m | n (A) = (x- m ■ ■ ■ X-i) d S^^ , 
where v = X' — (m d ) for some d > X\. Hence A_i > n if and only if v satisfies the 
condition in Theorem 13.151 when we have chL m | n (A) = cLf£T m | n (A). This completes 
the proof. □ 
For A G P+ |n with A = A (A), set 

(4.7) W Hn (A) = { w G W | £((v w >-)') < m, £^ w ' + ) < n }, 
where v = X' — (m d ) G Z^ with d > Ai. For w G W m | n (A), we define 

(4.8) w * A = - (d m )Y , v w ' + ) G Pi n> 
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where we identify P^i n with x Z™ (cf. (|4.3jl ). We can check that given w G 
W m | n (A), w G Wj^ m ^ for some q, and w * A does not depend on the choice of d. 
Now, we obtain a Weyl type formula for hook Schur polynomials, which recovers 
the Cheng and Zhang's formula [11] in a new combinatorial way. 

Theorem 4.3 (HH). For A G V + , , we have 

chL m|n (A) = Yl (-lY M ^K mln (w * A). 
wew m | n (A) 

Proof. By Lemma 14.11 and Theorem I3.10| we have 
chL m i n (A) 



(%— m ' ' ' 3?— 1 , 



E(-^w(y w )^- y (x- m]) n (i+^w 

\«>eW / iGf-ml 



el 



( E(-l)' M w(y[n]) s (—-)'-(^)( x p m] )) II (i + ^r 1 ^) 
\«iew / «e[-m] 

je[n] 

( S (-^^s^'+lyN)^^^- )'-(d-»))*( x hm]) J n + 

\iu£W / ie[-m] 

ie[n] 

E (-l)^chif(u;* A). 
weW m | n (A) 



□ 



Remark 4.4. Lemma 14.11 and hence Theorem 14.31 can also be proved from a Howe 
duality of (Ql m \ n , Qld) acting on the supersymmetric algebra generated by (C m l°* © 
C°l ra ) C d (cf.[?l[8]) without a combinatorial definition of chL m | n (A) by Berele and 
Regev. 

The denominator identity for fll m | n is given as follows. 
Corollary 4.5. For w G W m | n (0), letw*0 = (9™, 9™) G P+,. Then 
Y[(xi + yj)' 1 Y[(xi - x v ) Y[ (Vj ~ Vj') 

i,j i<i' j<j> 

EST (_^t{w 1 )+t(w 2 )+l{w)w 1 (e^+8 m )~nl m w 2 (e^+8 n ) 
/ 4 \ L > l-m] y [n] ' 

™G>V m | n (0) w 1 £S m 

where i,i' G [-m], j,f G [n], 5 m = Eie[- m ]H ~ fyiA = T,je[n]U ~ !) e j> x ™ = 
J2ie[-m] e i> an< ^ Srn (resp. S n ) is the symmetric group on the letters [—m] (resp. 

N). 
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Proof. Since the hook Schur polynomial corresponding to empty partition (that is, 
A = 0) is 1, the identity follows from Theorem 14.31 □ 

Remark 4.6. For w G W m i n (0), let \x be the corresponding partition (see Remark 
I3.3f) . Then u> * G P^\ n if an d only if £(fi) < m, n. Moreover, in this case, we have 

Since |//| = we may write by abuse of notation 
Yl( x i + ViT 1 Yl(xi - x v ) JJ (Vj - Vf) 



m\n 



h3 j<j 

f— iM^I f— 1 ^( m i)+^( m 2) x ^i(M*+5m)-nl m w 2 (fM+5 n ) 

£(fj,)<m,n w\£S m 

4.2. Representations of gl m+n . Let us consider a character formula for a cer- 
tain class of infinite dimensional highest weight representations of the Lie algebra 
0( m+ra , which comes from the study of unitary highest weight representations of 
its associated Lie group U(m,n) (cf.[15j [191 EH]) or horn a parabolic analogue of 
Kazhdan-Lusztig theory (cf.[6, 13J). 

For non-negative integers m and n, not both zero, let C m+n = C m © C n be the 
(m + n)-dimensional space, where { e,, | i € [—m] } (resp. { ej \ j G [n] }) is the basis 
of C m (resp. C n ). We may identify the general linear algebra g = Ql m+n with the 
set of (m + n) x (m + n) matrices whose row and column indices are from [— m] U [n] 
(or simply [— m, n]). 

We denote by f) and b the Cartan subalgebra of the diagonal matrices and the 
Borel subalgebra of the upper triangular matrices respectively. We also put t = 
gl m gl n , which is naturally embedded in g. Let P m +n be the Z-lattice of h* 
generated by { ej [ i G [—m,n] }, and -P^+n ^ ne se * °f t-dominant integral weights, 
that is, the set of weights A = ^ ie [_ mn j A^j G P m +n such that A_ m > ... > A_i 
and Ai > . . . > A n . 

Given A G P^ +n , let L°(A) be the finite dimensional irreducible highest weight 
t- module with highest weight A. We may view L°(A) as a representation of the 
parabolic subalgebra q = t + b , where the action is extended in a trivial way. Now 
we define the generalized Verma module V m +n{A) to be the induced representation 
V m+n (A) = U(g) <S>c/(q) L (A). Then V m+n (A) has a unique maximal irreducible 
quotient L m+n (A). Similarly, we define the characters of L m+n (A) and V m+n (A) in 
terms of the formal variables e A (A G P m + ra ). Put Xi = e €i for i G [— m, n]. Then 

, / a \ _ SA<o(x[- m ])g A >o(x w ) 

m+n( ) " n^(i-^-s-) ' 

where i G [-m], j G [n], A <0 = (A_ m , . . . , A_i) and A >0 = (Ai,... ,A n ). 
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From the Kazhdan-Lusztig conjecture [26] proved in [H |2] and its parabolic ana- 
logue [U [13] , we have 

(4.9) chL m+n (A) = ^2 b AA ,chV m+n (A'), 

where the integers 6a A' are determined explicitly in terms of (parabolic) Kazhdan- 
Lusztig polynomials evaluated at 1. 

Now, using Howe duality let us derive an explicit expression of chL m+n (A) for 
particular highest weights in P^ +n , which is equivalent to the Enright's formula 
given in [H]. For deN, let P d = © fce [ d ] %£k be the weight lattice of Ql d . Consider 
the symmetric algebra S? d generated by (C m ®C d )©(C n (g>C d ), where C m and C d 
are the duals of the natural representations of gi m and gl d respectively. Then there 
is a semi-simple (g[ m+n , g[ rf )-action on y d , which gives the following multiplicity free 
decomposition [191 ES] 

(4.10) y d = L m+n (A(A)) ® L d (X), 

where L d (X) is the irreducible highest weight gl^-module with highest weight A = 
(Ai, . . . , X d ) G 1% (or A = J2ke[d] G P d ), and A (A) is a highest weight in P+ +n . 
In fact, if 

(4.11) A = A+ + (A-)* = (A+ ...,A+ 0,...,0,-A-,...,-A^), 
then 

(4.12) A(A) = (-d, . . . , -d, -A" - d, . . . , -Ar - d, A+ . . . , A+, 0, . . . , 0), 

* v ' * v ' 

m n 

where we identify P m +n with Z m+n . 
Proposition 4.7. For A G we /iat>e 

(x_ m ---x_i) d chL m+n (A(A)) = 5A(x[„];x[_ m] ). 
Proof. The decomposition in (|4.10p gives the following identity 

(4.13) - ( [ X -Z' z \Tl ir l-^) = S chL m+n (A(A) )sA (z [d] ), 

where i £ [n], j 6 [— m], k G [d], and zj d ] = {zj, = e £k \ k G [d] }. Comparing with 
the Cauchy type identity in Theorem 13.101 it follows from the linear independence 
of rational Schur polynomials that (x_ m • • • X—\ ) d chL m+n (A(A))=4 nl/hm] . □ 

Corollary 4.8. For A G we have L m+n (A(X)) = V m+n (A(X)) if and only if 
d > n and A„ > 0. 
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Proof. It follows from Theorem 13. 151 □ 
We also have an interesting analogue of the Jacobi-Trudi formula from Proposition 

Corollary 4.9. For A G Z+, we have 

chL m+n (A(A)) = det(chL m+n (A(Ai - i + j)))i<ij<d- 

□ 

Suppose that A G Pm+n ^ s given and A = A(A) for some A G Z+. Set 

(4.14) >V m+ „(A) = { w G W | l(X w '-) < m, £(X W ' + ) < n }. 
For w G W m + n (A), we define 

(4.15) w * A = ((A— + (cP))*, A*'+) G P+ +n , 

where we identify Pm+ n w ith Z™ x Z™ . Now, we can state a Weyl type formula for 
chL m+n (A(A)) (A G 

Theorem 4.10 fcf.fH]). Given A G Pm+n wii/j A = A(A) for some X G Z+, we 

chL m+n (A)= £ (-l)^W m+n (™*A). 

«>ew m+n (A) 

Proof. It follows from Proposition 14.71 with Theorem 13. lUt The proof is similar to 
that of Theorem 14.31 □ 



Remark 4.11. The parametrization of highest weights for generalized Verma mod- 
ules in Theorem 14.101 is different from the one in [T5] , where the sum is given over 
the Weyl group of Ql m+n with its shifted action on the highest weight A, which is not 
equal to w * A for w G W m + n (A) by definition. It would be interesting to compare 
these two formulas. 

Note that we can observe a similarity between chL m+n (A(A)) and chL m | n (A(A)) 
(A G Z+) from Theorem 14.31 and Theorem 14.101 Recently, a more direct and deep 
connection between the Grothendieck groups of two module categories of gl m+n and 
flt m i„ has been found in [TO], which explains this similarity of characters. 

Example 4.12. Suppose that A = 0^. Then A(0rf) = (— dl m ,O n ) and each w G 
yVm+n(—dl m , n ) of minimal length corresponds to a unique partition \i C (m n ) with 
£(w) = \n\ (see Remark [33] (1)). Hence, we have w * (—dl m , n ) = (— (d m ) — //*, fj,), 
and 

chL m+n (-dl m ,0 n ) = 2^ (-l)M _ 1=1 , 

where i G [— m] and j G [n]. 
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Remark 4.13. We may apply Theorem 13.101 to other highest weight representa- 
tions of a Lie (super)algebras g, whenever we have a Howe duality of (g, gl d ) (cf. 
[3 [13 [13 [23] ) , since the associated irreducible characters for g satisfy a Cauchy 
type identity of the form given in Theorem 13.101 and hence they are equal to 
under suitable choices of A and B [28] . For example, S x >0 <0 yields a character of 
an integrable highest weight representation of 0^, which is an irreducible compo- 
nent in fermionic Fock space representations, and 5"^ >0//z<0 a character of a highest 
weight representation of gl^ (not integrable) appearing in bosonic Fock space rep- 
resentations (cf . [1T^ [23]). In both cases, the characters are given as an alternating 
sum of generalized Verma modules induced from integrable highest weight represen- 
tations of a parabolic subalgebra of gl^ (see Proposition 13.61 and 13. 7[) . We also have 
similar applications to super cases studied in [U El [29] (see also [28] ) . 
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